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We introduce the notion of topological entropy of a formal languages as the 
topological entropy of the minimal topological automaton accepting it. Using a 
characterization of this notion in terms of approximations of the Myhill-Nerode 
congruence relation, we are able to compute the topological entropies of certain 
example languages. Those examples suggest that the notion of a "simple" for¬ 
mal language coincides with the language having zero entropy. 


1 Introduction 

The Chomsky hierarchy classifies formal languages in levels of growing complexity. At 
its bottom it puts the class of regular languages, followed by context-free and context- 
sensitive languages. At the top of the hierarchy it lists the class of all decidable languages. 
As such, the Chomsky hierarchy gives a method to assign a measure of complexity to formal 
languages. 

However, using the Chomsky hierarchy as a mean to asses the complexity of a language 
has certain drawbacks. The most severe drawback is that the classification of the Chomsky 
hierarchy depends on a particular choice of computation models, namely finite automata, 
non-deterministic pushdown-automata, linear bounded automata, and Turing machines, 
respectively. It can be argued that this choice results in some contra-intuitive classifica¬ 
tions: of course, accepting a language to be "simple" as soon as it is accepted by a finite 
automaton is reasonable. The converse however is not: not every language that cannot be 
accepted by a finite automaton is necessarily "complicated". 

An example is the Dyck language D with one sort of parentheses [2]. This is the language 
of all words of balanced parentheses like (()()) and ((())) but not (())) or (. This language 
is context-free but not regular, and thus a classification by the Chomsky hierarchy would 
make this language D appear to be not so "simple". On the other hand, there is a very 
simple machine model accepting D, namely a two-state automaton with only one counter. 
It is reasonable to say that this kind of automaton is intuitively simple. The Chomsky 
hierarchy does not capture this: it puts the Dyck language with one sort of parentheses 
in the same class as much more complicated languages like palindromes. And there even 
exist context-sensitive languages that can be accepted by finite automata with only one 
counter. 
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To assess the complexity of a language one could now proceed as follows: given a language 
L, what is the simplest form of computation model that is required to accept L? It is clear 
that this approach heavily depends on the notion of "simplest computation model" and the 
fact that there is such one. Indeed, it requires a hierarchy of all conceivable computation 
models to make this approach work, an assumption that is hardly realizable. 

Instead of considering all possible computation models, we propose another approach, 
namely to consider one computation model that works/or every language. Then given a 
formal language L one could ask what the "simplest" instances of this particular compu¬ 
tation model is that is required to accept the given language L. This then can be used to 
assign to L a measure of complexity that does not depend on a particular a-priori choice of 
certain computational models. 

More precisely, we shall show in this work that we can use the notion of topological au¬ 
tomata [15] to assign to every formal languages a notion of entropy that naturally reflects 
the complexity of the formal languages. As such, we make use of the following facts: for 
every formal language there exists a topological automaton accepting it. Furthermore, for 
each topological automaton there exists a natural notion of a smallest automaton accept¬ 
ing the same language. Finally, as topological automata are a particular form of dynamical 
systems, we can naturally assign a measure of complexity to every topological automaton, 
namely its entropy. Therefore, we can define the complexity of a formal language L as the 
entropy of the minimal topological automaton accepting it. We call this notion the topological 
entropy of L. Intuitively, the lower the topological entropy of L the simpler it is. Languages 
with vanishing entropy are thus the simplest of all formal languages. 

An advantage of this approach is that it works for every formal language, and is thus inde¬ 
pendent of a particular choice of computation models. On the other hand, one could argue 
that this approach is purely theoretical, as it may not allow us to compute the entropy of 
formal languages easily. However, we shall show that it is indeed possible to compute the 
topological entropy for certain examples of languages. For this we use a characterization 
of the topological entropy in terms of approximations of its Myhill-Nerode congruence 
relation. Using this, it is not hard to show that all regular languages have entropy 0. More¬ 
over, we shall show that the Dyck language with one sort of parenthesis has also entropy 
0. Both of these can thus be called "simple", and intuitively they are. On the other hand, 
we shall also show that languages like palindromes or Dyck languages with multiple sorts 
of parentheses do not have zero entropy. 

The paper is structured as follows. We first introduce the notions of topological automata 
and entropy of semigroup actions and formally define the notion of topological entropy of for¬ 
mal languages. The main part of this paper is then devoted to proof a characterization 
of topological entropy that allows for a comparably easy way to compute it. This is done 
in Section 3. We compute the entropy of some example languages in Section 4. We also 
provide a characterization of the topological entropy in terms of the entropic dimension of 
suitable pseudo-ultrametric spaces. Finally, we shall summarize the results of this paper 
and sketch an outline of future work. 
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2 Topological Entropy of Formal Languages 


A variety of notions has been developed to assess different aspects of complexity of formal 
languages. Most of these notions have been devised with an understanding of complexity 
in mind that comes with classical complexity theory, and thus these notions are formu¬ 
lated as decision problems. Examples for this are the word problem and the equivalence 
problem for formal languages, and the complexity of the formal languages is measured 
by the complexity class for which these problems are complete. Other notions quantify 
complexity by other means. Examples are the state complexity [18] of a regular language, 
which gives the complexity of the language as the number of states in its minimal automa¬ 
ton, or the syntactic complexity of a regular language, which instead considers the size of 
corresponding syntactic semigroup [10]. 

The core idea of the present article is to expand the methods of measuring a formal lan¬ 
guage's complexity by a topological approach in terms of topological entropy, which proved 
tremendously useful to dynamical systems. Topological entropy was introduced by Adler 
et al. [1] for single homeomorphisms (or continuous transformations) on a compact Haus- 
dorff space. The literature provides several essentially different extensions of this concept 
for continuous group and semigroup actions. Among others, there is an approach towards 
topological entropy for continuous actions of finitely generated (pseudo-)groups due to 
Ghys et al. [13] (see also [3, 5, 7, 16]), which has also been investigated for continuous 
semigroup actions in [4, 6,14]. 

By a dynamical system we mean a continuous semigroup action on a compact Hausdorff 
topological space. Topological entropy measures the ability of an observer to distinguish 
between points of the dynamical system just by recognizing transitions at equal time inter¬ 
vals, i.e., with respect to a fixed generating system of transformations, starting from the 
initial state. Since the above notion of dynamical system may very well be regarded as 
the topological counterpart of a finite automaton, it seems natural to utilize the dynamical 
approach for applications to automata theory. 

To link dynamical systems to formal languages we shall use the already mentioned notion 
of a topological automaton [15]. This notion has been introduced as a topological general¬ 
ization of the usual notion of a finite automaton by allowing to have an infinite state space. 
Indeed, topological automata share certain properties with finite automata. Eor example, 
for each topological automaton there exists a minimal topological automaton accepting the 
same language. However, in contrast to finite automata, every formal language is accepted 
by some topological automaton, and not only regular languages. This allows us to uniformly 
treat all formal languages with one computation model. 

Recall that a (deterministic) automaton over an alphabet E is a tuple A = {Q,'E,,3,qo,F) 
consisting of a finite set Q of states, a transition function ti: Q x E —> Q, a set F Q Q of final 
states, and an initial state qo G Q. The transition function is usually extended to the set of 
all words over E by virtue of 


5*{q,£) := q, 

5*{q,wa) ■— 5{&{q,w),a) 
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for q ^ Q, a E IL, and w G E*. The language accepted by A is then 

C{A) := {w e E* I 5*{qo,w) G F}. 

It is not hard to see that the function 5* is a monoid action of E* on Q - indeed it is the 
unique monoid action of E* on Q extending d. 

The notion of deterministic finite automata can now be extended to an infinite state set 
as follows. Throughout this article a continuous action of a semigroup or monoid S on 
a topological space X is an action a of S on X such that fts: X —)■ X, iXs{x) = a{x,s) is 
continuous for every s E S. Note that the latter just means that a: X x S ^ X is continuous 
where S is endowed the discrete topology 

Definition 2.1 A topological automaton over an alphabet E is a tuple A = (X, E, a, xq/F) 
consisting of 

• a compact Hausdorff space X, called the set of states of A 

• a continuous action a ofL* on X, called the transition function of A, 

• a point xo E X, called the initial state of A, and 

• a clopen subset F C X, called the set of final states of A. 

We say that A is trim if (x{xq, E*) is dense in X. The language recognized by A is defined as 

C-{A) := {re G E* I a.{xo,w) E F}. 

Let B = (y, E, j6, yo, G) be another topological automaton. We shall say that A and B are isomor¬ 
phic, and write A = B, if there exists a homeomorphism cp: X ^ Y such that 

(p{a{x,cr)) =f{(p{x),cr) 

for all x E X, a E E, y>(xo) — yo/ (ind (p{F) = G. A 

Evidently, isomorphic automata accept the same language. 

Observe that every automaton accepting L can be turned into an automaton that is trim: if 
A = (X, E, oi, Xo, F) is a topological automaton accepting L, then replacing X with a(xo, E*) 
and F with F n a(xo, E*) always yields a trim automaton accepting the same language L. 

As already stated, and in contrast to regular languages, every formal language L C E* is 
accepted by a topological automaton, cf. [15, Proposition 2.1]. 

Proposition 2.2 Let L C E* and xl ihe characteristic function of L. Equip X := { 0,1 }^* with 
the product topology, and define the mapping S: X xX* ^ X by 

3{f,u){v) := f{uv). 

Then L is accepted by the topological automaton {X,'L,S,XL,T)for T := {f E X \ /(e) = 1 }. 
With the notation of 2.2, we define the minimal automaton ofL to be 

Ai = {xLfF^*),X,3,xuTi), 
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where Xl(^*) is the closure of in {0,1 and Tl = Tn;\;£(E*). Clearly, is trim. 

Indeed we have the following fact that justifies to call At minimal, cf. [15, Theorem 2.2], 

Proposition 2.3 Let L C E*, and let A = (X, E, vq, F) be a topological automaton accepting 
L. Then A = Al if and only if for every trim automaton B = {Y ,T,,yQ, \,G) accepting L there 
exists a uniquely determined surjective continuous function cp: Y ^ X satisfying (p{A{y,(7)) = 
5{(p{y),(j) and (p{yo) = vq. Moreover, in this case the unique cp satisfies G = (p^^(F). 

Since Al = Al, this proposition immediately yields that the minimal automaton is indeed 
minimal in the above sense. Moreover, in the case that L is regular, Al is finite and is the 
usual minimal automaton of regular languages. 

Example 2.4 Let E be a finite alphabet and let a,b E X, a b. We consider the Alexan- 
droff compactification Zoo of the discrete space of integers Z, that is the set Zoo = Z U {oo} 
equipped with the topology 

(M C Z U {oo} I 00 G M => Z \ M is finite}. 

We define an action a of E* on Zoo by setting OL{m,a) = m + 1, (x{m,b) = m — 1 and 
a(m, c) = m for all m G Zoo and c G E \ {a, b}. Then a constitutes a continuous action of E* 
on Zoo, and for each n G N the topological automaton A = (Zoo, E,a,0, {«}) accepts the 
language L = {zf G E* | \w\a = |zf |;, + n}. O 

We now shall express the complexity of the language L accepted by a topological automa¬ 
ton A = (Q, E, a, xq, F) by the topological entropy of the continuous action a of E* on Q [1, 
8,17]. To this end, we shall first fix some useful notation and recall some important defini¬ 
tions about continuous actions on compact Hausdorff spaces. 

Let X again be a compact Hausdorff space. We shall denote by C (X) the set of all finite open 
covers of X. If/: X ^ X is continuous and Ff G C(X), then:= {/^^(Lf) \ U eU} 
is a finite open cover of X as well. Given U,V E C{X), we say that V refines U and write 
W d Vif 

YV eV3U eU: V CU, 

and we say that U and V are refinement-equivalent and write U = V it U and V ^ Ff. 
Furthermore, if {Ui | z G J) is a finite family of finite open covers of X, then 

\JUi-.= {^Ui\{Ui)ieiEY[Ui]. 
iei iei !€/ 

is a finite open cover of X as well. For U E C{X) let 

N{U) :=inf{|V| | VCW,X = 1JV}. 

In preparation for some later considerations, let us recall the following basic observations. 

Remark 2.5 ([1]) Let X be a compact Hausdorff space, W, V G C(X), 7 be a finite set, 
{Ui)i(zi, (V,);gj G C(X)^, and /: X —)• X be a continuous map. Then the following state¬ 
ments hold: 
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\) U<v ^ N{U) < N{V), 

2) U<V ^ f-\U)<f-\V), 

3) (V/ el: Ui< Vi) \ji^,Ui < V,6i Vi- O 

Now we come to dynamical systems, i.e., continuous semigroup actions. Let S be a semi¬ 
group and consider a continuous action a of S on X. For U G C(X) we write 

s-HU) := a;\U). 


For every finite F C S and U G C{X) let 

{F:U), :=N{\/ s-\U)). 

seF 

Assume F to be a finite generating subset of S. If Ff is a finite open cover of X, then we 
define 

n—Foo ^ 

Furthermore, the topological entropy of a with respect to F is defined to be the quantity 

r]{oL,F) := sup{//(a,F,F/) | U G C(X)}. 

Of course, the precise value of this quantity depends on the choice of a finite generating 
system. However, we observe the following fact. 

Proposition 2.6 Let S be a semigroup and let a be a continuous action of S on some compact 
Hausdorff space X. Suppose E,F Q S to be finite subsets generating S. Then 


— ■ rj{oi,F) < rj{cc,E) < n ■ r]{a,F), 


m 


where m := ml{k G N | F C E^} and n := inf{A: G N | £ C F*^}. 


Proof Let U G C{X). Evidently, (E^ : U) < (F“ : U) for all A: G N, whence 

k^CO ^ k^co ^ 

log2(F'^VFf), ^ log2(F*^ : ZF), , ^ 

= nlimsup —— - < nlimsup — — — - = n ■ fj [a, F,u) 

k^oo k^oo ^ 

Thus, p{a,E,U) < n ■ y{a.,F,U). This shows that r]{a,E) < nr]{a,F). Due to symmetry, it 
follows that r]{cL,F) < m ■ r]{ix, E) as well. □ 


With all the necessary notions in place we are finally able to define our notion of entropy 
of formal languages. 

Definition 2.7 Let L C E* and let Al = (X,E, xo/^/h) be the minimal automaton of L. Then 
the entropy of L is the entropy E U { e }) ofS with respect fo E U { e }. A 
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3 A Characterization 


We claim that the definition of topological entropy is natural. Yet computing using the defi¬ 
nition alone may not work very well. It is the purpose of this section to remedy this issue by 
providing an alternative characterization of the topological entropy of formal languages. 
For this we exploit another way of considering formal languages as dynamical systems. 

To view a formal language L over an alphabet E as some kind of dynamical system we 
take inspiration from the characterization of regular languages as languages whose Myhill- 
Nerode congruence relation 0(L) has finite index. Recall that for m, p G E we have 

{u,v) G 0(T) \/iv G E*: {uw G L vw G L). 

The relation 0(L) can be seen as some way of measuring the complexity of L: if L is regular, 
the number of equivalence classes is finite and equals the number of states in the minimal 
automaton of L. Indeed, this is the idea behind the notion of state complexity. 

However, if L is not regular this measure is not available anymore. We shall remedy this by 
not considering the number of equivalence classes of 0(T), but by considering the growth 
of the number of equivalence classes of a particular approximation of 0(L). Based on this 
growth we introduce our characterization of topological entropy of L. 

Let us first recall some basic notation. Let 0 be an equivalence relation on a set Y. For 
y G y we put [yje := { x G Y | (x,y) G 0 }. Then Y/0 := { [y]© | y G Y }. Furthermore, 
the index of 0 on Y is defined as ind(0) := |Y/0|. For a mapping /: X —)■ Y we set 
/^^(0) := {{s,t) G XxX I G 0}. Clearly,/“^(0) then constitutes an 

equivalence relation on X. 

Now let E be an alphabet. The Nerode congruence of a language L C E* is the equivalence 
relation 

0(L) := {{u,v) G E* X E* I Vw G E*: MW G L G L}. 

Recall that L is regular if and only if it is accepted by an automaton. The following charac¬ 
terization of regular languages in terms of the Nerode congruence relation is well-known. 


Theorem 3.1 (Myhill-Nerode) Let E be a finite alphabet. A language L Q "L* is regular if and 
only if @(L) has finite index. 


Starting from this characterization we shall now make precise what we mean by approxi¬ 
mating the relation 0. For this we introduce another type of equivalence relation. 


Definition 3.2 LefLbe an alphabet. ForF C IL* finite and L C FL* define the function xf ,l'- 

{0,l}ffcy 




1 ifuw G L 
0 otherwise 


for u eYF and w G F. Now let 


©{F,L) := kerixFx)- 


A 
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Now, the equivalence relations 0(F, L) constitute an approximation of 0(F) in the sense 
that 

©(L) = f){0(F,L) I F C E* finite}. (1) 

Note that ind 0(F, L) = |im(xF,L) | < 2l^L In particular, 0(F, L) has finite index, and thus 
the following definition is reasonable. 

Therefore, as (0(E(”),L) | n G IN) may be regarded as an approximation of the Myhill- 
Nerode congruence 0(L), it seems natural to consider the exponential growth rate of the 
corresponding index sequence as a measure of complexity for a given formal language L. 
This motivates the following definition. 

Definition 3.3 Let E he an alphabet, and denote with J^(E*) the set of finite subsets ofL*. Define 
7 : J'(E*) X P(E*) ^ N, (F,L) ^ ind0(F,L). 


Given L QT,*, we call 

71,: J'(E*)^N, F^ 7 (F,L) 

the Myhill-Nerode complexity function of L. The Myhill-Nerode entropy of a language L C 
E* is defined to be 

h{L) := limsup 

n^oo 

where E(”) is the set of all words over E of length at most n. A 

The Myhill-Nerode complexity function of L has some immediate properties that we collect 
in the next proposition. 

Proposition 3.4 Let E be a finite alphabet, let E,F QT,* be finite, and let L, Lq, Li C E*. Then 

1 ) 7 (F, 0 ) = 7 (F,E*) = 1, and thus h{(Z)) = /z(E*) = 0. 

2) 7 (E U F, L) < 7 (E, L) • 7 (F, L). IfE C F, then j(E, L) < j(F, L). 

3) 7 (F,L) = j(F,Z*\L), and hence h(L) = h(Z*\L). 

4) 7 (F,Lo U Li) < j(F,Eo) ■ j{F,Li),and thus h{Lo U Li) < h{Eo) + h{Li). 

5) 7 (F, Lq n Li) < 7 (F, Lq) • 7 (F, Li), and thus hfLQ n Li) < /z(Lo) + L(Li). 

The goal of the rest of this section is now to show that the Myhill-Nerode complexity of 
L coincides with the topological entropy of L. We start this endeavor by showing that the 
Myhill-Nerode entropy of a formal language is bounded from above by the entropy of 
any topological automaton accepting it. In the case that the automaton is trim, these two 
notions even coincide. 

Theorem 3.5 Suppose A = (X, E, a, Xq, F) to be a topological automaton. Consider S := E U {e} 
and U := {F, X \ Fj. Then h{E{A)) < f]{a., S,U). If A is trim, then h{L{A)) = f]{K, S,U). 

We prove this theorem with the following three auxiliary statements. 


7l(X(>^)) 

n 
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Lemma 3.6 Let A = {X,'L,ci,xo,F) be a topological automaton. LefO: E* —)■ X, zf i—)■ a{xo,zv) 
and U := {F, X \ F}. Consider a finite subset E CX* as well as the equivalence relation 

Ae ■= {{^,y) I Vw G E: a{x,w) G F a{y,w) G F}. 


Then the following statements hold: 

1 ) x/AE = {y^eE^o-\u))\{0}. 

2) ©{EAiA)) = (OxO)-I(Ae). 

3) If A is trim, then 0(E*) nV ©for every V G X/Ag. 

Proof (1): We observe that V := (VzyeE^ ^(^)) \ {®} constitutes a finite partition of X 
into clopen subsets. For any f/ G V and x E V, we observe that 

[x]aj; = {y G Y I Vw G E: x{x,w) G F 4^ a(y,ze) G F} 

= {y G y I Vze G E: V G zf“^(F) 4^ y G ze“^(F)} 

= {y G y I Vze G E VLf eU : x E w~^{U) ^ y E ze“^(Lf)} 

= {yGy|VWGV:xGW4»yGW} 

= {y G y I y G y} 

= y 


We conclude that X/Af = V. 

(2): Let E := L{ A). For any two words u,v E E*, it follows that 


{u,v) E 0(E, E) 



Vze E E: uw E L vw E E 

Vze G E: a(xo/Mw) G F 44> a{xo,vw) E F 

Vze G E: a(a(xo,u),w) E F a(a(xo,v),w) E F 

Vze G E: a(0(M),ze) G F <44> a(0(z’),z<;) G F 

(0(m),0(z;)) G A£. 


Thatis,0(E,L) = (O x O)-I(Ae). 

(3): By (1), the set X/ Ae is a collection of open, non-empty subsets of X. If A is trim, then 
0(E*) is dense in X, and thus 0(E*) n y 7 ^ 0 for every V E X/Ae- □ 


Proposition 3.7 Eet A = (X, E, a, xq, F) be a topological automaton and let U := {F, X \ F}. 
Consider a finite subset E C E*. Then TL{A)i^) — • ^)a- Furthermore, if A is trim, then 

iL(A)i^) = (E : Ff)a- 

Proof Let E := L{A) and V := Vw6£(^~^(^))- Since V \ {0} constitutes a finite partition 
of X into clopen subsets, V \ { 0 } does not admit any proper subcover. Consequently, 
N(V) = |V\ {0}|. Applying 3.6, we conclude 

7l(E) = |EV0(E,L)| |X/Ae| |V\ {0}| = N(V) = (E : 


Finally, if A is trim, then 3.6 (3) asserts |E*/0(E, E) | = |X/Ae | and therefore 7 l(E) = (E : 

W)^. □ 
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The particular choice of the cover U = {f, X\f} seems arbitrary, but this is not the case. 
Indeed, if the automaton A= (Q, L, a, xq, F) is minimal, then the entropy t]{a,'LLI {e}) of 
the automaton equals {e },U)- We shall show this fact in 3.10. As a preparation, 

we shall first investigate three auxiliary statements. 

Lemma 3.8 Let X be a set, let S be a semigroup, and let a: S x X ^ X be an action of S on X. 
Let U be a finite cover ofX and let M,N Q S be finite. Then 

V s-\u)^ y 

seMN seN teM 

In particular, the complexities of those two covers coincide. 

Proof Without loss of generality we may assume that U is closed under intersection: in fact, 
U is refinement-equivalent to the finite cover U '.= {{^V \V QU}. Hence, if the desired 
statement was true for U, then this would imply 

V s-\U)= V s-\U) = \J s-WJ t-\U)) = \J s-y\J t-\U)) 

seMN seMN seN teM seN teM 

due to the statements (2) and (3) of Remark 2.5. 

Henceforth, assume that U is closed under intersection. We shall show an even stronger 
claim, namely 

V s-\u)= y s-yy t-\u)). (i) 

seMN seN teM 

To ease readability, let us denote the left-hand side by C, and the right-hand side by TZ. 

Let Y E C. Then 

y= n ^^HUs) 

seMN 

for some {Ug \ s G MN) G ns6MN®^^(^)- each s G MN we can choose Tg E M, Vs E N 
such that s = TgCTg. Then 

y= n ^^HUs) 

seMN 

= n {Tcrgr\Ug) 

seMN 

= n 

seMN 

= n n (^^yr-\Ura)) 

creN reM 

= n ‘"'Hn r-\Ura))en 

creN reM 


Conversely, let Y ElZ. Then 

’p= n n 

creN reM 
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for some {Ua,T \ cr e N,t e M) e n(£r,T)6MxN Then 

creN reM 

= n n {^^)~Hua,T) 

creN reM 

= n S^^(n{Ua,T\cr ^N,T e M,S = T0-}^ 

seMN 

Define 

Us := P|{D£r,T \ (7 E N,T e M,S = Td}. 

Then Us E U for each s E MN, as U is closed under intersections. But then 

y= fl s-\Us)ec 

seMN 

as required. 

Finally, Equation 2 and Remark 2.5 (1) yield 

N{ V s-\u))=n[\/ s-^iy rHu))), 

seMN seN teM 

as it has been claimed. □ 

Lemma 3.9 Let L Q T,* and let A = (X,E, a, xo/f) Ihe minimal automaton of L. Consider 
S := S U {e} and U := {F, X \ F}. IfV is a finite open cover ofX, then there exists some n G N 
such that V ^ VseS" ^ -\U). 

Proof For n E N, let us consider the equivalence relation 

A„ := Aj.(„) = { (x,y) G X X X I Vw E : oc{w,x) E F a{w,y) G F } 

(cf. Lemma 3.6). We are going to show that 

W := {[x]a„ I n G N, X G X, 3y G V: [x]a„ C R} 

is an open cover of X. By Lemma 3.6 (1), it follows that W is a collection of open subsets 
of X. Thus, we only need to argue that X = IJ W. To this end, let x G X. Since V 
is a cover of X, there exists some V E V with x G V". As 1/ is open in X with respect 
to the subspace topology inherited from {0,1}^*, we find a finite set E C E* such that 
W {y G X I Vw G £: x(w) = y(w)} C V. Let n G N where E C S". We observe that 

[x]a„ = {y G X I Vw G S”: a(x,w) G F a(y,w) G F} 

= {y G X I Vw G S”: a(x,w)(e) = 1 a(y, w)(£) = 1} 

= {y G X I Vw G S”: x(w) = 1 y(w) = 1} 

= {y G X I Vw G S": x(w) = y(w)} 

C W C y. 
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Accordingly, [x]a„ S W and hence a: G IJ W. This proves the claim. Now, since X is 
compact, there exists a finite subset Wo where X = IJ Wq. Due to finiteness of Wo, there is 
some n G N such that Wo A XfA„. We conclude that 

V A w A Wo A X/A„ \ ^ (Vses-' 

which completes the proof. □ 

We finally reached the point where we can show that the Myhill-Nerode complexity and 
the topological entropy of L coincide. 

Theorem 3.10 Let L C E* and let A = (X, E, a, Xq, F) be the minimal automaton ofL. Consider 
S := Eu{e} andU := {F, X\F}. Thenh{L) = //(a,S,ZF) = 7/(a,S). 

Proof Define U := { F, X \ F }. Since A is trim, we know that h(L) = t]{a, S,U) by Theo¬ 
rem 3.5 and hence h{L) < rj{(x,S). To show the converse inequality, let V be a finite open 
cover of X. We show that//(a,S,V) < t]{u,S,U). According to 3.9, there exists some 
m G N such that V is refined by VseS™ ^ ^ (^)- Then 

<-'(«») = 

by 3.8. Now we obtain 


,(«. s. V) = limsup '.sMLlAk itasup 'sMAAIA 




Therefore, ^(tt, S) < rj[oL,S,U) and hence//(a, S) = rj{oL,S,U) =h{L) hy 3.5. 


□ 


4 Examples 

3.10 allows us to easily compute the topological entropy of certain classes of languages. 
To begin with, we show that all regular languages have zero entropy. 

Theorem 4.1 Let E be an alphabet and L C E*. The following are equivalent: 

1) Lis regular, 

2) is bounded, and 

3) there exists some finite subset F C E* such that 0(F, L) = 0(L). 

Proof 1) => 2). Due to 3.1, 0(L) has finite index. Note that 0(L) C 0(F, L) and hence 
7 l(F) < ind 0(F) for all F C E* finite. Thus, is bounded. 

2) =4> 3). Suppose that 7 l is bounded. Then there exists some finite Fq C E* such that 
7l(Fo) = sup{7l(F) I F C E* finite }. We shall show that 0(Fo, L) = 0(F). 0f course, 
0(F) C 0(Fo, F). Let (m, u) G (E* x E*) \ 0(F). By (1) there exists some finite Fi C E* such 


12 



that {u, v) i. 0(Fi, L). Obviously, Fq U Fi C E* is finite and 0(Fo U Fi, L) C 0(Fo, L). By 
assumption, 7l(Fo U Fi) < 7l(Fo). Consequently, 0(Fo U Fi, L) = 0(Fo, L) and therefore 
{u,v) e (E* X E*) \0(Fi,L) C (E* X E*) \0(FoUFi,L) = (E* x E*) \ 0(Fo, L). This 
substantiates that 0(Fo, L) = 0(F). 

3) => 1). By assumption 0(F) = 0(F, F), and since 0(F, F) has finite index, 0(F) has 
finite index as well. Hence, F is regular due to 3.1. □ 

Corollary 4.2 Let E be an alphabet. IfLQ'L* is regular, then h{L) = 0. 


The converse of this corollary does not hold, i.e., there are non-regular languages with 
vanishing topological entropy. To see this we shall show that Dyck languages always have 
zero entropy (cf. 4.8). We shall put the corresponding argumentation in a more general 
framework, by estimating the entropy of languages defined by groups. For this purpose, 
we recall the concept of growth in groups. Consider a finitely generated group G. Let S be 
a finite symmetric generating subset of G containing the neutral element. The exponential 
growth rate of G with respect to S is defined to be 

egr(G,S) := limsup 

n^co ^ 


Note that this quantity is finite as | < \S\^ for every n G IN. Furthermore, 

egr(G,S) = lim 

n-^oo n 

due to a well-known result by Fekete [12]. 0f course, the precise value of the exponential 
growth rate depends upon the particular choice of a generating set. 

However, if T is another finite symmetric generating subset of G containing the neutral 
element, then 

i-egr(G,T)<egr(G,S)<Z.egr(G,T) 

where k := inf{m G IN \ {0} | T C S'”} and I := inf{m G IN \ {0} | S C T'”}. This justifies 
the following definition: G is said to have sub-exponential growth if egr(G, S) = 0 for some 
(and thus any) symmetric generating set S of G containing the neutral element. The class of 
finitely generated groups with sub-exponential growth encompasses all finitely generated 
abelian groups. In fact, if G is abelian, then 


S” c 


{n^ 

ses 


a(s) 


a: S 


{0. n}] 


and thus |S” | < (n -|-1) for all n G IN. Now let us return to formal languages. 


Theorem 4.3 Let E be an alphabet. Let Gbe a group, (p: E* —)• G fl homomorphism, H Q G, and 
E C G finite. Define 


P(p{H) := {w G E* I Vm prefix of w: (p{u) G H}, 
F^(H,£) :=P,p{H)ncp-\E). 
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Then' y{F,L^{H,E)) < \E\ ■ \cp{F)\ + 1 for all finite F QT,*. In particular, 
h{L^{H,E)) < <log2|S|. 

>0O ^ 

Furthermore, if S is a finite symmetric generating subset of G containing the neutral element and 
k := inf{ra G N \ {0} | C S'”}, then 

/2(4(H,E)) </c-egr(G,S). 


Proof We abbreviate P := Pip{lT) and L := Lq,{H, E). Consider a finite subset F C E*. Then 
Q := E(p{F)~^ is a finite subset of G. Fix any object oo ^ Q and define Qoo := Q U {oo}. Let 
us consider the map i/j : E* —Qoo given by 


ip{u) 


(p{u) if M G pn f ^(Q), 
oo otherwise 


(m G E*). 


We show ker xp C ©(F, L). To this end, let (w, v) G ker xp. We proceed by case analysis. 

First case: xp{u) = xp{v) f oo. Now, m, p G Pn(p^^(Q) and (p(u) = xp{u) = xp{v) = (p{v). 
Let w G F and suppose that uzv E L. We show vw E L. We observe that 


(p{vzv) = (p{v)(p{w) = (p(u)(p(w) = (p{uw) E E, 


i.e., vw E (p^^{E). In order to prove that vw E P, let x be a prefix of vw. If x is a prefix 
of V, then (p{x) E H as v E P. Otherwise, there exists a prefix y oi w such that x = vy, 
and so we conclude that cp{x) = (p{vy) = (p(v)(p(y) = (p{u)(p{y) = (p{uy) E H, because 
uw E P and uy is a prefix of uw. Hence, vw E E. On account of symmetry, it follows that 
{u,v) E @{F,L). 

Second case: xp(u) = xp(v) = oo. Let x G {u, v}. If x ^ (p^^{Q), then we conclude that 
(p{xw) = (p{x)(p{w) i E and thus xw ^ L for any zf G F. If x ^ P, then xw ^ P and hence 
xw ^ L for any w E F. This proves that {uw, vw} n L = 0 for all ze G P. Consequently, 
{u,v) E ©{F,L). 

This substantiates that kerxp C ©(P, L). Therefore 

7 (P,L) = ind©(F,L) <ind(kert/;) < |Q«.| < |Q|+1 < |E| • |<p(F)|+L 


In particular, it follows that 

ML) = < Itasup lofo(|£|>(^<-)l+l) 

n^co ^ >-co ^ 

, iogj®(E(''))i , iog,(|sr) , 

= limsup - - < hmsup ^ = log 2 |E|. 

n—^-oo ^ n—T-oo ^ 

Finally, suppose S to be a finite symmetric generating subset of G containing the neutral 
element. Since E is finite, M := {m G IN \ {0} | (pijG) C S'”} is not empty. Let k := inf M. 
Our considerations above now readily imply that 

HL,(S, E)) < limsup < L. Hmsup = Ic ■ egr(G, S), □ 

n^co ^ n^oo ^ 
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For groups whose growth is sub-exponential the previous theorem yields that the corre¬ 
sponding languages L^(S, E) have zero entropy 

Corollary 4.4 Let S be an alphabet, let Gbea group with sub-exponential growth, and 
a homomorphism. Then for each S Q G and finite E Q G, it is true that h{L,p{S, E)) = 0. 

We immediately obtain the following statement. 

Corollary 4.5 Let E be an alphabet, let G be a finitely generated abelian group, and cp: YT ^ G a 
homomorphism. Then for each S C G and finite E Q G, it is true that h{L,p{S, E)) =0. 

The following corollaries are immediate consequences of Theorem 4.3 for S = G. 

Corollary 4.6 Let E be a finite alphabet and L C E*. Let G be a group, (p: Y* ^ G a homomor¬ 
phism and E C G finite such that L = (p^^{E). Then j{F,L) < |E| ■ |^(E)| -|- 1/or all finite 
F QY*. In particular. 


h{E) < limsup —— 

n—>-oo 

Corollary 4.7 Let Y be a finite alphabet, L C E*. Let G be an abelian group, cp-.Y* G a 
homomorphism and E C G finite such that L = (p^^{E). Then h{L) = 0. 

With the previous results in place, we are now able to argue that Dyck languages have fi¬ 
nite entropy. Recall that the Dyck language with k sorts of parentheses consists of all balanced 
strings over { (i, )i,..., (jt, )k }■ Alternatively, we can view the Dyck language with k sorts 
of parentheses as the set of all strings that can be reduced to the empty word by succes¬ 
sively eliminating matching pairs of parentheses. 

We can formalize this as follows. Let E, E be two alphabets, A := E U E, and let x: E —> E 
be a bijection. Consider the the free group E(E) with generator set E, and denote with 
cp: A* —>• E(E) the unique homomorphism satisfying cp{a) = a and cp{K{a)) = a~^ for all 
fl G E. Define 


|<P(eW)| 


< log2 1^1 


D(k) := {w G A* I (p{w) = e A (Vm prefix of w : \u\a > \u\K(^a))}- 

If E = { (i,..., (fc }, E = {)i,... ,)k}, and = )„ then the set D(k) coincides with the 
Dyck language with k sorts of parentheses. 

Theorem 4.8 Let k: Y ^Ybe a bijection between finite sets. Then 

log/El <E(D(k)) <log2(2|E|-l) 
for S := E U E^^ U {e}, where e denotes the neutral element o/E(E). 

Proof Let E := D{k). We first show the inequality ind ©(E^'^/ E) > \Y"\, since this implies 
log 2 |E| < h(p{K)). For this let M, u G Y'^,u v. Define k(m) := K(M|y|)... a:(mi), where 
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u = Ui... M|„|. Then u • k{u) G L, but v ■ k(u) ^ L. Thus (u,v) ^ and therefore 

ind0(E(”),L) > |E" I as required. 


For the second inequality let us consider the unique homomorphism ip: F(E) ^ 'Zp satis¬ 
fying 


ip{b){a) 


1 if a = b, 

0 otherwise 


{a,b G E). 


We observe D{k) = {e}), where the mapping cp is as above. Hence, we have 

h{D{K)) < egr(F(E),S) by 4.5. As it is known that egr(F(E),S) = log 2 ( 2 |E| — 1) we 
obtain the claim. □ 


Note that for |E| =1 we have h{D{K)) = 0. Thus D(k) is an example of a non-regular 
language with zero entropy. For |E| > 1 the exact value of h{D{K)) is unknown to the 
authors. 

The reason that Dyck languages with more than one type of parentheses have non-zero 
positive entropy is the following: the different types of parentheses occurring in a word 
w G D{k) need to be mutually balanced, i.e., (p{w) = e. In other words, if we replace 
this requirement by the weaker condition that each opening parenthesis has to be closed 
eventually, then we obtain a class of languages with zero entropy. 

Theorem 4.9 Let k: IL ^ T, be a bijection between finite sets, let A := E U E, and consider the 
language 

D'{k) := {w G a* I Vfl G E: {\w\a = \MK{a)) ^ prefix ofw : \u\a > 

Then h{D'{K)) = 0. 

Proof Let us consider the homomorphism cp: A* ^ given by 

(p{w){a) := \w\a — \MK{a) S A*, a G E). 

We observe that D(k) = L^(N^, {0}), wherefore h{D{K)) = 0 by 4.3. □ 

Other non-regular languages with vanishing entropy are discussed in the following exam¬ 
ples. 

Example 4.10 Let E be an alphabet. 

1) Let m G N and a, b G E, a 7 ^ fc. Then L {w G E* | \w\a = \w\}, + m} is not 
regular. However, h{L) = 0 by Corollary 4.7. To see this, note that the mapping 
cp: T,* ^ Z, w \w\a — \w\b constitutes a homomorphism where L = (p^^{{m}). 

2) Suppose E = {a,b,c}. Then L := {a’^b"‘c’^ \ m G N } is not context-free, but 
h{L) = 0. To see this we show that for every n the relation 0 = 0(E(”), L) has the 
equivalence classes 

k < nil 

l<£<k, 2k — £<n 
l<£<k, k — £<n 

[b]e. 
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From this it follows ind L) G 0{rp-), and thus h.{L) = 0. 

To see that the sets in (3) are indeed all equivalence classes of L), let u eT.* 

such that u is not an element of the first three types of classes in (3). We need to show 
that then u G [fc]©( 2 ;(n) We do this by showing that there is no w G E^'^) such that 
uw G L. 

Assume by contradiction that such a word w exists. Then iv must be of one of the 
following forms 

w = a^h^c^, 2k + £ < n, 0 < £ < k, 
w = b^c^, k + £<n, 0<£<k, 
w = c^, 0 < £ < n 

\i w = a^b^c^, 2k + £ < n, 0 < £ < k, then u = , k — £ < n/2, and therefore 

u G ly a contradiction. If w = b^c^, k + £ < n,0 < £ < k, then u = 

and k — £ > 0, 2k — {k — £) < n, thus u G ©(£(«) ly, again a contradiction. 

If w = c^, then u = and k — {k — £) < n, so u G [a^b’^c^]Q(^^(„) another 

contradiction. 

Thus, our assumption that w exists is false. The same is true for the word b, and thus 
u G [b]@(^-^{n)^iy as required. O 

Next we are looking for an example of a language with non-zero entropy. Of course, by 
what we have already shown, a suitable candidate for this has to be non-regular. But do 
not have to require much more: the following example shows that there are context-free 
languages with non-zero entropy. 

Example 4.11 Suppose |E| >2. Then the palindrome language 

L := {ww^ I w G E*} 

is not regular, but context-free, and h{L) G (0, oo). 

To see h{L) > 0, observe that for each n G N and all u,v E E”, if (m, z^) G ©(E^'^), L), then 
u = V. This is because if vv^ E L, we also have uv^ E L, and hence u = v. Thus 

[^]0(E('>),L) [^]0(E("),L) 

Thus ind ©(E(”), L) > |E'’| = |E|'’, and we obtain 

loe lEl” 

h{L) = limsup- - -= log 2 |E| > 0. 

ft—>oo ^ 

To see h{L) < oo we shall consider the relation ©* defined by 

{u,v) E ©* {u,v) E ©(e(”^,L) and (|m| < n |u| < n). 

Then ind ©(E^”), L) < ind ©*. We shall show 

log2(ind©*) 

limsup —-- < 00. 
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There are at most many equivalence classes [u]@* for u G E*, \u\ < n. To count the 
number of equivalence classes for \u\ > n we define 

£n{u) := { fli... fl, I 1 < / < G E,M = fli... uiu',u' G L }. 


Then for u,v eT,* \ E^”) we have 

(m,u)G0* (m,u) G 0(e(”),L) (4) 

The first equivalence is clear. To see the second equivalence let (u,v) G 0(E(”),L), and 
let fli ... fl,- G in{u). By definition of i„{u) it is then true that u{ai... G L. Because 
{u, v) G 0(E(”), L) we therefore obtain v{ai... G L, i.e., v is of the form v = ai... Ujv' 
for some v' G L. This yields fli... fl, G Ini'll)- By symmetry we obtain i„{u) = £„{v) as 
required. 

Conversely, assume in{u) = £n{,'v), and let w G E^'^) be such that uw G L. Because \u\ > n, 
there exists u' E L with uw = w^u'w. Then E in{u) = £n{v), and therefore v = w^v' 
for some v' E L. But then vw E L. By symmetry vw E L =4> uw E L for each w E E(”), 
and therefore (m, v) E 0(e('’), L), as required. 


Using the characterization from Equation (4) we have 



|{4(m) I M gE*\eW}| 


Now every set £„ (u) with u = u-[ .. .Uk,k > n, can be represented by the prefix mi ... of 
u of length n together with a tuple t E { 0,1}” defined by 

ti = 1 Ui - . .Ui E £n{u). 


Therefore, 


This yields 


and thus 


= \{in{u) I M GE*\eW}| < |E|'^-2” 


ind 0* = 




/s* 


< |e(”)| + lEl” •2'^, 


, log,(ind 0*) , , i^ix 

limsup —- < log 2 ( 2 |E|) < 00 . 




It is unclear to the authors whether the upper bound obtained in the proof of 4.11 is related 
to the one in 4.8. 

It is not hard to see that the entropy of a formal language can very well be infinite. This is 
illustrated by the following example. 

Example 4.12 Let |E| >2, and choose mappings (p„: T?" -E ^(E”) for each n G N such 
that |im(^„)| = |Ep" = 2^”. Then define a language L C E* by 

j {uv \ u E T?-" ,v E q)n{u)} if m = 2” + n for some n G N, 

0 otherwise. 
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Then 2^" < 


i}" < ind0(E”,L). 


(5) 


To see this we shall show that each word (pn{u) defines its own equivalence class, i.e., for 
words Uq,Ui G T?" with (p„{uo) ^ (pn{ui) we have (mq, Wi) i 0(E”, L). This is because 
if (pn{uo) 7 ^ fn{u-i) we can assume without loss of generality that there exists some word 

V G (pniuo) \ (p„(ui). By definition of L we then have uqv G L, but since \uiv\ = 2” + n and 

V ^ ^n(Mi) we also get UiV ^ L. Thus (mq/Mi) ^ 0(E”, L). 

But then (5) implies 


li^ toscnW > li^ 1^ ^ 

n—>-00 ft—>oo 

and thus h{L) = oo. 




5 Topological entropy and entropic dimension 


Another interesting characterization of the entropy of formal languages is in terms of the 
entropic dimension of a suitable precompact pseudo-ultrametric space. For this recall that 
a pseudo-metric space {X,d) is called precompact if for each r G (0, oo) there exists some 
finite set F C X such that 

X = U{Brf(v,r) I vGF}. 


If (X, d) is a precompact pseudo-metric space, then define 


7iX4)ir) ■= inf{ |F| | F C X finite,X = [J{B,{x,r) | x G F } }. 


Then the entropic dimension dim(X, d) of the precompact pseudo-metric space (X, d) is de¬ 
fined as [11] 


dim(X, d) := limsup 


log2(T(x,i)(?')) 

log 2 (l/r) 


To now obtain a precompact pseudo-metric space (X, d) whose entropic dimension is the 
same as the topological entropy of a given language L, we shall first start with a general 
observation. Let X be a non-empty set and let 0 = (0„ | n G N) be a descending sequence 
of equivalence relations on X. Define d©: X x X —>• [0, oo) as 

d©(x,y) := 2-“f("eN|(x,y)^0„} ^ 


It is easy to see that dQ{x,x) =0 and dQ{x,y) = dQ{y,x) is true for all x,y G X. Moreover, 
as 


{n G N I (x,z) ^0„}C{nGN| (x,y) ^ 0„ V (y,z) 

= {n G N I (x,y) ^0„}U{nGN| (y,z) t }, 

we also have d@{x,z) < max{ d©(x,y),d©(y,z) } for all x,y,z G X. Because of this (X, d©) 
is a pseudo-ultrametric space. 


19 



Proposition 5.1 Let X be a non-empty set and let 0 = (0„ | n ^ IN) be a descending sequence 
of equivalence relations on X such that each has finite index in X. Then {X,d@ ) is precompact 
and 

dim(X, d©) = limsup ——— 

It—>oo ^ 


Proof We first observe that for all x, y G X and n G N 

d@{x,y) < 2^" n < inf{ ra G N | {x,y) ^ 0m } ^ ©n- 


Therefore, X/0„ 
and 


{B^q{x,2 ”) I X G X}. Since X/0„ is finite, (X,d©) is precompact 

7(x,rfe)(2^”) = |X/®«l- 


Consequently, 


as required. 


dim(X, d©) = limsup 
)-^0+ 

= lim sup 

n^co 

= lim sup 

n^co 


log2(7(x,rfe)(0) 

log2(l/r) 

log2(7(x,rfe)(2~'')) 

n 

log 2 |X/ 0 „| 

n 


□ 


A straightforward application of this lemma is the following theorem. 

Corollary 5.2 Let E be an alphabet and let L C E*. Then with 0 := (0(e(”), L) | n G N) 

dim(E*,d 0 ) = h{L). 


In the case that the language L is represented by a topological automaton we obtain the 
following result. 

Theorem 5.3 Let A = (X, E,a, xq, F) be a topological automaton. Let A = {A„ \ n G N) where 

A„ := Aj.(„) = { (x, y) G X X X I \/w G E*^”^ : a-{w, x) G F x{w, y) G F } 

whenever n G N (cf. 3.6). Then h{L{A)) < dim(X,dA). Furthermore, if A is trim, then 
h{L{A)) = dim(X, dA). 

Proof Let L := L{A) and n G N. We observe that 7 l(E(”)) = |E*/0(E(”),L)| < XfA„ by 
3.6 (2). Moreover, if A is trim, then = X/A„ due to 3.6 (3). Hence, 5.1 yields the 

desired statements. □ 
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The pseudo-metric considered in the theorem above does not necessarily generate the 
topology of the respective automaton. In fact, this happens to be true if and only if the au¬ 
tomaton is minimal, i.e., isomorphic to the minimal automaton of the accepted language. 
Furthermore, this case can be characterized in terms of a separation property: a topological 
automaton is minimal if and only if the induced pseudo-metric is a metric. 

Proposition 5.4 Let A = (X,L,a,Xo,F) be a topological automaton and L := L{A). Then the 
topology generated by d^ is contained in the topology ofX. Furthermore, the following statements 
are equivalent: 

1) A^A{L). 

2) d/^ is a metric. 

3) generates the topology of X. 


Proof By 3.6 (1), the subset {x, e) = [^] is open in X for all v G X and e G (0, oo). 

Hence, the topology generated by d/^ is contained in the original topology of X. Now let 
us prove the claimed equivalences: 

(2) =4>(3): Suppose that d/^ is a metric. Then the topology generated by dj^ is a Hausdorff 
topology. Since this topology is contained in the compact topology of X, both topologies 
coincide due to a basic result from set-theoretic topology (see [9, §9.4, Corollary 3]). 

(3) =4>(1): Assume that dj^ generates the topology of X. This clearly implies d/^ to be a 
metric. Consider the unique surjective continuous homomorphism cp: A ^ A{L). We are 
going to show that f is injective. To this end, let x,i/ G X such that (p{x) = (p{y). We argue 
that d/^{x,y) = 0. Let n G IN. For every w G we observe that 

a{x,w) G F (p{k{x,w)) G Ti S{cp{x),w) G Tl 

d((p(y),w) G Tl cp{oc{y,w)) G Ti a{y,w) G F. 

Thus, (x,y) G A„. It follows that {x,y) G flneN^n hence dA{x,y) = 0. Since d^ is 
a metric, we conclude that x = y. Accordingly, is a bijective continuous map between 
compact Hausdorff spaces and therefore a homeomorphism. This again is due to an ele¬ 
mentary result from set-theoretic topology (see [9, §9.4, Corollary 2]). 

(1)=4>(2): Suppose cp: A ^ to be the necessarily unique isomorphism. Concerning 
any two points x, y G X, we observe that 

(x, y) G A„ (A) Vw G E^”) : oc{x,w) E F oc{y,w) ^ F 

Vw G E^”): (p{oi{x,w)) G Tl <S=> y>(a(y,w)) G Ti 

Vw G E^”^ : 3{(p{x), w) E Tl 5{(p{y), w) G Ti 

« {cp{x),(p{y)) G An{A{L)) 

for every n G N. Hence, dA(^)(x,y) = dA(j{(i)){(p{x),(p{y)) for all x,y G X. Accord¬ 
ingly, it suffices to show that dA(yt(L)) is a metric. To this end, let f,gE XiiX*) such that 

dA(A(L)){f'S) = 0- ^0 argue that f = g- Let w G E*. Then there exists n G N where 

w G Since dA(yt(L))(//S') = 0/ conclude that {f,g) G A„{A{L)) and thus 


f{w) = 1 


3{f,w){e) = 1 S{f,w) G Tl 

5{g,w) G Tl « d{g,w){e) = 1 « g{w) = 1. 
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Therefore, f{zv) = giw). It follows that f = g- This shows that rfA(yt(L)) ^ metric and 
hence completes the proof. □ 


6 Conclusions 

In this paper we have introduced the notion of topological entropy of formal languages as 
the topological entropy of the minimal topological automaton accepting it. We have shown 
that this notion is equal to the Myhill-Nerode complexity of the language, and can also be 
characterized in terms of the entropic dimension of suitable pseudo-ultrametric spaces. 
Using these characterizations, we were able to compute the topological entropy of certain 
types of languages. 

The main motivation of this work was the goal to uniformly assess the complexity of for¬ 
mal languages independent of a particular collection of computation models. We believe 
that the examples we have provided in this work show that the notion of topological en¬ 
tropy of formal languages is a suitable candidate for such a complexity measure. In par¬ 
ticular, we have shown that some languages intuitively considered to be simple all have 
zero entropy: regular languages, Dyck languages with one sort of parentheses, our "com¬ 
mutative" version of Dyck languages with arbitrary sorts of parentheses, and the language 
{ I n G N }. Indeed, all of these languages are accepted by simple models of com¬ 

putation, e.g., one-way finite automata with a fixed number of counters. 

On the other hand, we have presented examples of languages that have non-zero entropy 
that can hardly be considered as simple, namely Dyck languages with more than one sort of 
parentheses as well as the palindrome languages. Indeed, palindromes cannot be accepted 
by deterministic pushdown automata, and Dyck languages with more than one sort of 
parentheses give rise to the hardest context-free languages [2]. 

A natural next step in investigating the notion of topological entropy is to provide more 
examples that test the suitability of this notion as a measure of complexity of formal lan¬ 
guages. For example, we have already shown that all languages accepted by finite au¬ 
tomata have zero entropy. A natural question is now to ask for which classes of compu¬ 
tation models the topological entropy of the accepted languages is also zero. We suspect 
that this is the case for one-way finite automata equipped with a fixed number of counters 
and an acceptance condition that does only require to check local conditions, including the 
current values of the counters. 

Conversely, one could ask what properties languages with non-zero entropy possess. What 
form of non-locality in a suitable machine model is necessary to accept such languages, 
given that they are decidable? And what properties do languages have if their topological 
entropy is infinite? Are there context-free languages with infinite entropy? 

Acknowledgments: We would like to express our sincere gratitude towards the anonymous 
reviewer for numerous valuable suggestions that significantly improved the presentation 
of the paper. 
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